J2ee Technology Performance Evaluation Methodology

Internet-enabled enterprise information systems using middleware technology are becoming increasingly popular. These technologies are becoming more and more complex. However, there is little understanding in the software industry on the strengths and weaknesses of competing technologies. This paper describes the approach being taken in CSIRO's Middleware Technology Evaluation (MTE) project to attempt to alleviate some of these problems. In evaluating application server performance, we have used a simple benchmark application in preference to the more complete benchmarks, such as ECperfTM or RUBiS. This benchmark has the dual advantage of supplying unambiguous results, focussed on a particular area of J2EE and the flexibility to test different J2EE programming idioms. Performance evaluation requires care in the choice of methodology. We have been able to improve performance by more than an order of magnitude by adjusting the tuning parameters available in some application servers. We have followed the methodology reported here consistently to produce such effects.